Skip to content

Backport(v1.19) buffer, in_http: enforce size limits on decompressed payloads#5393

Merged
kenhys merged 1 commit into
v1.19from
backport/in_forward_buffer
Jun 25, 2026
Merged

Backport(v1.19) buffer, in_http: enforce size limits on decompressed payloads#5393
kenhys merged 1 commit into
v1.19from
backport/in_forward_buffer

Conversation

@Watson1978

Copy link
Copy Markdown
Contributor

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
This PR introduces a strict upper bound for decompressed payloads.

Changes

  1. Introduced Fluent::Plugin::Extractor
    • Extracted the decompression logic (gzip, zstd, deflate) into a dedicated module that processes and decompresses data in manageable 64KB chunks, ensuring steady memory usage.
  2. Added decompression_size_limit parameter (default: 256MiB)
    • Specifies the upper limit of decompression in buffer (buf_file, buf_file_single and buf_memory) and in_http plugins.
    • If the limit is exceeded, it safely raises Fluent::Plugin::Extractor::SizeLimitError (or returns a 400 Bad Request in in_http).

Notice for 3rd-party Plugin Developers

Plugins utilizing the buffer / in_http mechanisms or the newly introduced Fluent::Plugin::Extractor are automatically protected by this limit.

However, if your custom plugin implements its own decompression logic independently, you must take care to implement similar boundary checks to guard against unexpected memory spikes.

Docs Changes:

Release Note:

  • buffer, in_http: enforce size limits on decompressed payloads

…payloads

**Which issue(s) this PR fixes**:
Fixes #

**What this PR does / why we need it**:
This PR introduces a strict upper bound for decompressed payloads.

### Changes

1. Introduced `Fluent::Plugin::Extractor`
   * Extracted the decompression logic (gzip, zstd, deflate) into a dedicated module that processes and decompresses data in manageable 64KB chunks, ensuring steady memory usage.
2. Added `decompression_size_limit` parameter (default: `256MiB`)
   * Specifies the upper limit of decompression in `buffer` (`buf_file`, `buf_file_single` and `buf_memory`) and `in_http` plugins.
   * If the limit is exceeded, it safely raises `Fluent::Plugin::Extractor::SizeLimitError` (or returns a `400 Bad Request` in `in_http`).

### Notice for 3rd-party Plugin Developers
Plugins utilizing the `buffer` / `in_http` mechanisms or the newly introduced `Fluent::Plugin::Extractor` are automatically protected by this limit.

However, if your custom plugin implements its own decompression logic independently, you must take care to implement similar boundary checks to guard against unexpected memory spikes.

**Docs Changes**:

**Release Note**:
* buffer, in_http: enforce size limits on decompressed payloads

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978 Watson1978 requested a review from kenhys June 25, 2026 07:06
@kenhys kenhys merged commit f5f2b7c into v1.19 Jun 25, 2026
13 of 19 checks passed
@kenhys kenhys deleted the backport/in_forward_buffer branch June 25, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants